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(57) ABSTRACT 

Distortion correction in an imaging system using parametric 
motion estimation. Parametric motion estimation is used to 
correct for geometric lens distortion and illumination 
changes. Models for geometric lens distortion and different 
illumination models are presented, as is a method for an 
iterative linearising method of solving the resulting sets of 
nonlinear equations. 
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CORRECTING DISTORTION IN AN 
IMAGING SYSTEM USING PARAMETRIC 
MOTION ESTIMATION 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention deals with the correction of distor- 
tion in imaging systems, more particularly to systems such 
as scanners and in cameras used in applications such as 
image capture and video conferencing. 

2. Art Background 

With the advent of inexpensive imaging sensors, appli- 
cations such as video conferencing and remote image cap- 
ture are becoming more prevalent. As manufacturers provide 
imaging systems at lower and lower price points, compro- 
mises are made especially in the optics provided, leading to 
systems having geometric lens distortion. As these systems 
are placed in more locations, problems with illumination 
arise, both as a result of variations in lighting conditions, as 
well as the effect of automatic brightness controls in the 
imaging systems themselves. 

The field of motion estimation deals with processing 
images for example to stitch together successive overlapping 
images to form a unified mosaic. These motion estimation 
techniques deal with arrays of pixels, and operate by deriv- 
ing the motion between a group of pixels in one frame and 
the same pixels in a new position in a second frame. For 
motion estimation to succeed, the illumination levels and 
therefore pixel intensity between the two frames must be the 
same, and the frames must be relatively free of geometric 
distortion. What is needed is a way to correct for the effects 
of geometric lens distortion in motion estimation, and to 
correct for changes in illumination that effect pixel density. 

SUMMARY OF THE INVENTION 

Motion estimation techniques and the Optical Flow Equa- 
tion (OFE) are applied to correct for distortion in an imaging 
system. Geometric lens distortion is corrected using a novel 
optical correction model. Illumination changes are corrected 
using both simple and Vignetting models. The solutions 
involve solving of a set of nonlinear equations. Typical 
solutions to these equations are computationally expensive. 
An iterative linearising solution is presented. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is described with respect to par- 
ticular exemplary embodiments thereof and reference is 
made to the drawings in which: 

FIG. 1 is a block diagram of an imaging system, 
FIG. 2 is a diagram showing geometric lens distortion, 
and 

FIG. 3 is a flowchart of the linearized solution. 

DETAILED DESCRIPTION 

Inexpensive imaging sensors arc enabling more applica- 
tions of imaging technologies, such as video conferencing 
and image scanning. Compromises in sensor design, and in 
sensor placement lead to distortions in the resulting images. 

Motion estimation techniques deal with processing 
images for example to stitch together successive overlapping 
images to form a unified mosaic; this technique us used for 
example in assembling scanned images of a large area using 
multiple passes of an imaging sensor. Motion estimation 
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allows the overlaps to be recognized and the smaller images 
firom each pass stitched together to form the larger unified 
mosaic. Similar techniques also arise in videoconferencing 
when the videoconferencing camera is moved, or a video- 

5 conferencing camera is used to scan a document. 

To succeed, motion estimation presupposes the illumina- 
tion and therefore pixel densities of overlapping image 
portions to be the same, and that the images are free from 
geometric distortion. Inexpensive optics introduce geomet- 

10 ric distortion, and may also introduce illumination differ- 
ences such as vignetting, the gradual fall off of illumination 
or image density away from the central axis of the image 
sensor. Additional illumination distortions are introduced by 
automatic brightness controls in image sensors, and by poor 

15 or uneven illumination in the environment. 

Two dimensional motion estimation refers to determining 
the displacements or velocities of pixels from one frame to 
another. The motion is induced as the same physical point on 
the surface of an object in 3-D space is mapped to different 
locations within the imaging plane at different time. In the 
sampled image domain this results in different pixel indices. 
This difference in pixel indices taken relative to one of the 
frames is called the "motion vecUDr" at that pixel between 
those two frames. Each pixel in the image has a motion 

25 vector of dimension two representing horizontal and vertical 
displacements. The collection of all motion vectors is some- 
times called the "motion vector field", or simply "motion 
field". The term "field" should be understood in a broader 
sense, not in strict mathematical definition of "vector field". 
Motion estimation from sequences of images is discussed 
for example in A. M. Tckalp, Digital Video Processing, 
Prentice Hall Signal Processing Series; M. I. Sezan and R. 
L. Lagendijk, Motion Analysis and Image Sequence 
Processing, Norwell, Mass.: Kluwer, 1993; and J. K. Aggar- 

^5 wal and N. Nandhakumar, "On the computation of motion 
from sequences of images- A review," Proc. IEEE, vol. 76, 
pp. 917-935, August 1988. 

FIG. 1 shows a system for performing motion estimation. 
Bus 100 interconnects image capture device 110 with central 
processing unit (CPU) 120, Memory hierarchy 130 provides 
storage for instructions and data, including storage of frames 
if data from information capture device 110, typically con- 
taining a mixture of RAM, ROM, and permanent disc 
storage. Not shown are other typical items such as displays 
and network connections. Image capture device 110 may be 
a device such as a television camera with appropriate frame 
capture hardware, or other imaging device such as a tele- 
conferencing camera, CCD image sensor, scanner, or the 
like. 

50 

Optical Flow 

In principle, all pixels within an image can be thought to 
follow a given trajectory through time. The trajectory of a 
given pixel can be parameterized via time, given an initial 
position at some reference time. If we take the reference 
time as t-t^, then 

where c denotes the trajectory followed by a pixel originat- 
60 ing at (x,y,ti). For the development that follows, we abbre- 
viate x(t;x,y,ti) as x(t), and apply similar abbreviations for 
y(t) and c(t). Even though physical conditions generally 
limit the variation between trajectories of neighboring 
pixels, conditions such as occlusion and uncovered back- 
65 ground can lead to arbitrary (independent) trajectories 
amongst neighboring pixels. In this treatment we neglect 
occlusion and uncovered background. 
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Assuming further Ihe intensity/color value of a pixel does 
not change with time along the motion trajectory, then, with 
I(x,y,t) denoting the intensity function. 



d/jcU)) 

dt 



= 0. 



(1) 5 



Using the chain rule of vector calculus. Equation (1) can 
be written as 

dl{c{i)) _ a/U,y. 0 dxjt) ^ y, i) dyjt) ^ diix, Q _ ^ (2) 
di ~ dx dt dy IT dt dt ~ 

Letting 

iyix, y. t) = 



dx 

dy 
dlix, y,t) 



dt 

results in 



10 



15 



(3) 



(5) 



The second method of proceeding from (5) is to approxi- 
mate the derivatives involved by first order differences and 
then multiply through by (t2-ii)- Then, once again, if we 
have just two frames aitj and i^, we can discretely estimate 
U and using either frame, and the interpretation of this 
amounts to whether we employ a forward or backward 
approximation to estimate the derivatives. 

In the development to this point it has been assumed that 
there is an explicit temporal evolution of the image intensity 
distribution. To enforce this assumption the chain rule has 
been applied to the intensity distribution temporal deriva- 
tive. We can equivalently, however, begin from a completely 
discrete perspective and proceed instead by applying a 2-D 
spatial Taylor series approximation. To that end, assume 
from the onset that we have two captured frames at times t^ 
and tj. Now, the notion of intensity invariance is expressed 
as (ignoring illumination change effects): 



20 



(10) 



25 



(4) 



where and 1^ are the spatial image gradients, 1^ is the 
temporal image gradient, and where x' and y' are the 
horizontal and vertical velocity components respectively, 
Equation (4) is called the optic- flow-equation (OFE), some- 
times abbreviated as: 



where (xi,yj) refers to the image intensity at the pixel 
(xj.yj in the first frame captured at time t^, and (^z^Vt) 
refers to the intensity at the point (x^^y^ in the second frame 
captured at time Note that (x^,y^) are integer valued, but 
(X2»y2) real-valued. Both Ij and 1 2 are assumed to be 
continuous and differentiable functions with respect to x and 
y. Using the definition of motion vectors, one can write 



(11) 



We can proceed using (4) in two equivalent ways. In the 
first approach integrate both sides of (4) from t^ to t2with 
respect to t, and assume both I^(x,y,t) and I^(x,y,t) are 
constant with respect to time to obtain: 

IJx,)^t)ixit:d-x{t,))^f,ix,y,t)iy(t^^^^^ (6) 



35 



where u(.,.) and v(.,.) are the horizontal and vertical motion 
vectors, respectively. Combining Equations (10) and (11), 
and using a Taylor Series Expansion of differentiable 
functions, 



hiJCu yi) = h(.JCi +u{xuyil yi+\ixi,yi)) 
^h{x,y)\ 



(12) 



Stiix. y)\ 



v(x\, yi) + h.o.i 



For convenience, define: 

f2{x,y)=I{x,y,t^, h(x,y)^I(x,y, O, I2,ix,y>l2(x,y)-f,(x,y) (7) 

and 

uix,y)=x{t^-x{t,\ v{x,y)MQ-y{ti), (8) 
to obtain 



fMy>ix,y)+I^x,y')v{x,y)-i-J2i(x,y)-0, 



As a result of assuming I^ and are constant with respect 
to lime between Ij and tj, we drop time dependence from the 
notation. Second, this assumption taken together with the 
continuity of intensity assumption and (1) imphes that I, 
(x,y,t) is constant between tj and l^. This allows for trivial 
integration of I, (x,y,i) on this interval. Lastly, estimates of 

and \y can be obtained anywhere on [ii,t2]. This is 
important to note because if we begin with just two frames 
at t^ and t^, we can discretely estimate I^ and using either 
frame. 



The Taylor series is expanded around the pixel (Xi,y^). 
5° Assuming either that u(Xi,yi) and v(xi,yi) arc small quan- 
tities or that the higher order terms are all zero, we can 
disregard the higher order terms in Equation (12), and arrive 
at 



55 



(9) 



y) \ 
sy [ 



yix\, yi) + 



(13) 



(/2(Jfi.yi)-/i(Jfi,>'i))aiO. 



60 



Note that by employing the same notation as in the previous 
development we write 



65 ^^(■«iy)«C-«,y)+/y(j£iy)v(j^y)+/2iM«o, 

which is the same result (9) previously arrived at. 



(14) 
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Parametric Motion Models 

In many realistic imaging scenarios a 2-D motion field is 
induced by either camera motion or objects in the 3-D world 
undergoing a rigid motion. Each of these efifects produces a 
smoothly varying 2-D motion field (provided that the depth 
of the objects docs not change swiftly in proportion to its 
distance to imaging device). Noting the fact that the motion 
field is smoothly varying, there have been attempts to 
describe it using a parametric model. Namely,the motion 
vector at any pixel needs to obey a model characterized by 
a few parameters. Therefore, the motion vector of a pixel is 
completely represented by the model parameters and its 
location. 

Given a vector of model parameters a the motion is 
expressed as u(x,y)=f(x,y;a) and v(x,y)=g(x,y;a). In order to 
estimate the model parameters we define the following cost 
function, 



5* 



Of T-^ 



07) 



?1 
dv 



Equation (17) can be re -written in the matrix form as 
follows: 



(18) 













15 
















[::l= 
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(15) Equation (18) can be solved to estimate global displacement 
parameters. 



The parameter estimation procedure is to take derivatives 
of with respect to each element of a, set those derivatives 
to zero, then solve the resulting set of equations. For 
non-linear models f and g a linearization step is typically 
employed. 

In the following, we will examine the parametric motion 
models: 1) translational motion, 2) aflBne motion, 3) bilinear 
motion, and 4) perspective motion. These parametric motion 
models can be shown to be the projection of the motion of 
simple 3-D objects, such as a plane, under different projec- 
tion models onto the 2-D imaging plane. Lastly, (15) is 
extended to include a-priori information and apply the result 
to estimate constrained global translational motion, 

Translational Motion Estimation 

One of the simplest models that can be imposed on the 
motion vector field is that all the motion vectors be the same, 
i.e, every pixel in the image is moving in the same direction 
by the same amount. This motion model is usually called 
"global translational motion". However, it should be noted 
that translational motion does not necessarily mean that 
objects in the 3-D (real) world translate. 3-D translational 
motion could only be reduced to a 2-D translational model 
under the assumption that the 3-D object is a planar object, 
and an orthographic camera model is utilized. Additionally, 
global translational motion can be induced by a camera 
translation while a "distant*' scene is being imaged. It may 
sound a very restrictive motion model, however there arc 
applications to which this model perfectly fits. An example 
is multi-frame resolution enhancement technologies for 
scanner applications, where multiple scans of a flat page are 
taken such that a translational shift occurs between each 
scan. 

The motion model constraints for this case are 

<^yh%v{x,yyv^ (16) 

where u^ and v^ refer to global horizontal and vertical 
displacements, ften, taking the derivatives of (15)with 
respect to u and v, and setting them equal to zero yields 



25 

Affine Motion 



30 AflSne motion is another commonly employed motion 
model. It can account for 2-D transformations such as global 
rotation, translation, zoom and skew. Physically, affine 
motion is generated between two images when the 3-D 
scene is planar and two images are acquired using different 
imaging plane positions under the orthogonal projection. 
The affine motion model is described by 

40 



i v(x,y)«a4X+asyi-a6 (19) 



where a, model parameters (sometimes 

called "Affine Parameters") and (x,y) is the pixel location. 
50 Note that global translational motion results when a^, Z2, 84 
and a5 are all zero. 

To solve for optimal affine parameters in the Least Square 
55 sense, we expand (15) as 

* = ^ t/xU. y)ixai + ^03 -H as) -t- Iy{x, y)ixa4 +^05+06) + fiiix, y)]^ 

60 

As previously mentioned, tjj is minimized by taking the 
derivatives with respect to the affine parameters ai,a2,a3,a4, 
65 a5,a6, and setting each derivative equal to zero. After car- 
rying out the algebra, one can arrive at the following linear 
matrix equation. 
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Bilinear Motion 

The global bilinear transform is used to improve on the 
capabilities of the aflfine transform, yet still yield a linear 
estimation problem. In the case of an affiae transformation 
parallel lines always remain parallel. This is not the case for 
motion induced by a perspective -projection imaging model 
The perspective model, however, which will be treated next, 
yields a non-linear problem. The bilinear transform can be a 
good compromise in that it transforms horizontal and ver- 
tical straight lines into straight lines, but does not produce 
parallel lines for any given parallel pair lines. The potential 
problem in using a bilinear transform is that lines that are not 
horizontal or vertical will be transformed to have some 
degree of curvatme. If the curvature becomes severe the 
transform may not be representative of the actual motion and 
imaging system. 

The derivation of global bilinear motion model estimation 
is quite similar to afifine motion model estimation. In 
particular, the motion field is governed by: 



Perspective Motion 

TTie perspective motion model is the most versatile of the 
20 motion models used thus far. Physically,the perspective 
model takes into account the 2-D mapping induced among 
perspective projected images of a plane in 3-D under dif- 
ferent focal plane positions. A complication is that the 
perspective model is nonlinear in its model parameters. 

The motion field in global perspective model is governed 
by the following equations. 



25 



tt(x, y) = 
v(j:, y) = 



ojx + a^y + 1 
07x4- a8>'+ 1 



(21) 



Before using these model parameters in (15) they are first 
35 linearized around the default perspective parameter a =(1,0, 
0,0,1,0,0,0) to yield 



[20) 40 



v(pc,y)»a^A-ia^-l)yA-a^-ai 



(22) 



By substituting (20) into (15) and setting the derivatives 
with respect to model parameters equal to zero, we obtain, 



To solve for optimal perspective motion model parameters 
in the Least Square sense, we apply (22) to (15) and obtain 
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arrive at a linear matrix equation for optimal motion param- 
eter estimation. Although any diflferentiable cost function 
would be acceptable, the model parameter solution may 
otherwise require costly non-linear optimization techniques. 
^ In particular, the constraint cost function is in the form 



To minimize ^, we take the derivatives of W with respect 
to perspective parameters aj, a^, a3, a^, aj, a^, a^, and ag and 
set them equal to zero. After carrying out the algebra, one 
can arrive at the following linear matrix equation. 



(26) 
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where 



Then, Equation (25) can be re-written as 



35 



40 



x,yEl x,yet 
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-ZfV2i+6) 



(27) 



The solution of this simple 2 by 2 matrix equation yields the 
optimal translational motion model parameters and v^ 
under the given quadratic constraint cost function. Of par- 
ticular importance is the cost function 



Note that we have utilized the Taylor Series Approxima- 
tion twice, in contrast to aL previous derivations. Until this 
point, only linearization of the intensity function has been 
required. However, we had to additionally linearize the 
motion model to derive the Linear Matrix Equation form of 
Equation (23). Due to this added linearization step, perspec- 
tive parameter estimation can be less accurate and stable. 

In some applications, one may have a-priori information 4>(M,v)-(K-aJ^+Cv-v^^ (28) 

relating to the motion model parameters. Such information, 

for instance, could be a probabihty distribution of the model where we know the expected translational parameters, 
parameters. We take this additional information into account 

by modifying the cost function (15) such that Intensity Changes 

All methods above are derived under the premise that the 
(24) intensity of a pixel does not change along the motion 
trajectory. As explained earlier, this may not always be true. 
The external illumination may change and the gain/offset of 
the image capture device may vary between captured 
frames. To take these effects into account, we modify 
Equation (10) such that 



2i(/,M+/^V + /,)^+^(«. V)} 



Where the a-priori information is embodied using (t>(u,v). 

We will only derive the equations to estimate the glol3al ^5 
translational motion model parameters under such con- 
straints. In this case we proceed as before and obtain 



Kl/j(j^i»yi)+K2-^2(jf2iy2) 



(29) 



a* 



(25) 



60 



dv 



65 



At this point, assume that the constraint cost function is a 
quadratic function in u and v. This assumption is made to 



From here, the optimization proceeds exactly as before, 
except that parameters and need to be estimated as 
well. We shall now derive the affine motion model parameter 
estimation equations under the assumption (29). Derivations 
for other motion models arc not repeated since the outline of 
the derivation is very similar in each case. 

Incorporating the affine motion model equations given by 
the Equation (19), into (29), we write: 



(30) 
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From which Taylor Series expansion results in: Using Equation (33) and (34), we write 



To solve for the unknown parameters, define a cost function ^ 
^ as below: 



10 



Putting Equation (35) into Equation (33) yields 



Now take the partial derivatives of V with respect to the y^/trf — + ill + yf- + 1 = o 

aflSne parameters ai,a2,a3,a4,a5,aQ and the illumination I Ui Jj ^ 

change model parameters K^.tCj, and set them equal to zero. 
After carrying out the algebraic steps, we obtain. 
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Lens Distortion 

Inexpensive lens systems introduce geometric distortion 
into the imaging system. This is shown in FIG. 2. Geometric 
distortion causes the point shown in undistorted first frame 
210 to appear in a slightly different location in geometrically 
distorted image 220. This same distortion causes the appar- 
ent motion of this point in undistorted second frame 230 to 
appear instead at the position shown in distorted second 
frame 240. 

In this section, we will develop methods of motion 
estimation between images captured with an imperfect lens 
that causes geometric distortions. The gamma factor and 
external illumination changes are also taken into account. A 
nonlinear optimization framework to solve for distortion 
parameters is introduced. Linearization of this model results 
in OFE-like equations. 

The geometric lens distortions can be modeled as 



(33) 
(34) 



There are two solutions to this quadratic equation: 



40 



2^H^T3|) ■ 



(37) 



45 As Kj goes to 0, y should approach y^. Since, 

I+^l-4*f7(^|+>|) 



■yg = 



50 



55 



this can not be a solution. On the other hand, the second 
solution satisfies this requirement, hence it is the true 
solution. 



Una 



\-p-AK-,{x]^-y]) 



-yt^yx 



yi = 



1 + Ki{x^ + y'^) 

y 



60 



where K7 is the lens geometric distortion parameter, (x^y^ 
are the distorted pixel positions and (x,y) are the corrected 
pixel positions. 



There are many parametric motion models that map pixels 
from one frame to another. The afifine model is used for 
subsequent derivation in this section. The translational, 
55 bilinear, and perspective motion models follow exactly the 
same procedure, and are therefore not treated for the sake of 
space. 
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Using the geometric distortion model given by Equation 
(33), we write 



= 3,(*l, *f2, *3, ir,, «6, Kj, Xg, yg) ~ 



(38) 



(39) 



10 



where 



(40) 



where x(x^yp and y(x^,y^) are as given by Equation (37). 

Therefore, we have a nonlinear mapping from one frame 
to another characterized by 7 parameters. With the assump- 
tion that intensity does not change along the motion 
trajectory, we write: 



Nonlinear Geometric Correction 
We can define a cost function given by 



(41). 



(42) 



This is a function of the af&ne and lens-distortion 
parameters, 

A variant of Powell's method for multivariable nonlinear 
optimization is used to find the parameters that minimize the 
cost function. Other multivariable nonlinear optimization 
methods known in the art may also be used. To speed up the 
calculation, rather than summing over all x^,y^ e Ij, sum- 
mation over several small rectangular regions in the image 
may be performed. The algorithm's performance depends on 
the size, location, and number of regions used. The accuracy 
of the estimate may be improved by choosing regions with 
features such as corners, rather than uniform regions, and by 
choosing regions towards the periphery of the image (where 
warping is more pronounced). Nonlinear optimization meth- 
ods also benefit from a good initial guess. The affine-motion 
estimation method is used on the middle portions of the 
images (where the warping is relatively mUd) for initial 
estimates of the afi&ne parameters. 

The widely used lens distortion model given by x^=x(l+ 
K7(x^+y^)) is not only a cruder approximation of the physics 
of the lens, but also give rise to more complicated equations, 
which requires the calculation of roots of a cubic polyno- 
mial. 

A hybrid system that alternates between the affine-motion 
estimation method to estimate the six affine motion 
parameters, and a one-dimensional nonlinear optimization 
technique to estimate the lens-distortion parameter is used 
for optimization. For the one-dimensional optimization, the 
"Golden Search" method is used to minimize the cost 
function *P, here considered as a function of the single 
parameter Ky with the affine parameters (Kj,iC2,K3,JC4,K5,Kg) 
clamped. 



14 

Linearization in Geometric Correction 

We may also use a Taylor Series approximation to get a 
set of linear equations with a closed-form solution. This 
process is shown in FIG. 3. Once the nonlinear model is 
selected 310, giving a system of nonlinear equations, an 
initial set of parameters is selected 320. Given the set of 
parameters, the system of nonlinear equations is linearized 
around the set of parameters 330, resulting in a linear 
system. Tbis linear system of equations is solved 340 using 
standard techniques to reach a new set of parameters. The 
process of linearizing the system of nonlinear equations 
around the set of parameters and resolving the resulting 
15 linear solution is iterated 350 until our convergence criteria 
is met, or for a set number of steps. The linearization 330 is 
done as shown previously by Taylor Series expansion. If we 
were to center the Taylor Series around the parameter set 
krf^y=(l,0,0,0,l,O,0) as we have done in previous sections, we 
would gel no information about the lens-distortion param- 
eter K7, since. 



20 



25 



--— =Oand =0. 



This is expected, since kj^y=(l,O,O,O,l,0,O) induces zero 
motion, and with no motion, both images are the same, so we 
have no hope of recovering the lens-distortion parameter. 
Likewise, a Taylor Series approximation around a 
parameter-set corresponding to rotation around the center of 
the image provides no information on the lens distortion, as 
35 the distortion is radial. But other affine-parameter sets (k^, 
K2,K3,K4,K5,Ke) yield an observable system for the lens- 
distortion parameter. More precisely, a Taylor Series 
approximation of l2(x'^y'g) around the parameters K^-(Ki, 
K2,K3,K4,K5,K6, 0) would yield 



40 



(43) 



where 



50 



55 



60 
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a3, 1 

{-Tkik^ki - 3klk2~k2ki ^k2)x\yg + 

i-3kikl +ki -Tkik^ks -k,kl)xgyl + 

(_A3 +k2-k2kl)y^^ + (~3k\ki-7J:ik4k6-kikl)x^g ^ 

(-6/£jM3 - 2^2*4*6 -2A3M3 -2Aiitji6)jf^>'g 4- 

i-3ki4-kikl-'2J(ik4ke)x,+ 

i-k2kl - 3k2kl - '2k,kMyu + (-4 -hkl) 
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<53i 1 



(53, 



53:1 ^-.| , 



53,1 53,1 



=(-A4Af-A3^-A4)J^ + 
(-A5^2 +A5)>J - 3*5*6 -2A4*iA3)xJ + 

(-6*4*6*5 - 2*4<:3*2 - 2*s*i*3 - l*^*,*!^,^, 4- 
(-2*5*3*2 -M?-3MJ}>J 4- 
(-2*6*1*3 -3*4*a-*4*3V,+ 
(-3*^*5 -2*6*3*2 -M3)>i +(-^6 -Ml) 



10 



and Iw=l2(^»y)-li(xH^yH-)t with aod the images of x 
and y under the transfornialion given by k^-(Ki,K2,K3,K4, 

^5.^6,0). 



Illumination Correction 

With the incorporation of external illumination change 
J 5 and oSset variation in capture devices as explained earlier, 
we modify Equation (41) as 



(44) 



20 and we redefine the cost function in terms of motion, lens 
distortion, and illumination change parameters as: 

The method described earlier is used for an initial esti- 
mate of the affine parameters, which are then used as the 2 ('f8/i(jc,.>,)+*f9 -/2(Jt;, y'^yf (^^5) 
center of the Taylor series expansion. If the initial estimate 's-yg^'i 
is almost entirely translational, one may expand the Taylor 
series around just the translational component of the initial 

estimate to simplify the compulations, taking k^-<l,0,K3, '^o account for vignetting effect, the vignetting correction 

0,l,Kg,0). This gives function is introduced, and Equation (44) modified as 



33, I 
^1/ 



(46) 



-3k3;cJ - U^x^y, - AjjyJ - (*2 + 3fc|)jf^ - 2*3*^:^, - (-A| ^ kjkl) 



euid 



35 



where 



a3, 1 

— = -k^x"^ - 2kix,yg - 3k6y] - Ik^kiXg - (3*| + *|)>, - (*| i- kek^) 



nxg, yg) = yiocos^[arctan[ ^''"^^^" |j + (1 - 



but may give inaccurate results if there is a substantial 40 

non-translational aspect to the motion. "^^^^^ ^^^^^ ^^^^^^ focal length, and (x^,y J is the camera 

Substituting Equation (43) into Equation (42), and taking imaging plane coordinates of the pixel (x^y^). 
the derivatives and setting ibem equal to zero yields the Then we can define a cost function in terms of (Ki,K2, 
following linear matrix equation: ^3,1^4, K5,K6,K7,K8,K9,»Cio) as 
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7. The method of claim 1 where the method of solving the 
fio'it^/' ys) ^Xii fi^^r y's^^ ^"^^^ ^* nonlinear equations is a multivariablc nonlinear opti- 

l H'(jr,,^,) vn^^73^J mization method. 

8. The method of claim 1 wherein step b) further com- 
^ prises: 

This cost function can again be minimized via non-Unear selecting an initial set of solution parameters; 

optimization techniques, or we can perform Taylor Scries . , 

approximation to arrive at a set of linear equations, which hneanzmg the set of nonlmear equations around the set 

can easily be solved. of solution parameters; 

The foregoing detailed description of the present inven- ^° c) solving the resulting linearized set of equations to reach 
don is provided for the purpose of illustration and is not an updated set of solution parameters; and 

intended to be exhaustive or to limit the invention to the ^ ^^^^ ^ ^^^-^^^ ^^^^ by a 

precise embodunents disclosed. Accordingly the scope of , ^ r c - ^ c • • / 

ihe present invention is defined by the ap^nded claims. ^^^^^ ^ number of iterations and a 

What is claimed is: 15 predetermined convergence catena is met, 

1. A method for correcting distortion in an imaging method of claim 8 where the set of nonUnear 
system, the method comprising: equations includes correction for geometric lens distortion. 

a) allying a motion estimation model to a first and a claim 8 where the set of nonfinear 
second image to generate a set of nonlinear equations equations includes correction for illumination changes, 
representing distortion, wherein the motion estimation 20 l^- method of claim 8 where the set of nonlinear 
model accommodates a change in illumination along at equations includes correction for both geometric lens dis- 
least one motion trajectory between the first and second tortion and illumination changes. 

images; 12. The method of claim 11 where the model for illumi- 

b) solving the set of nonlinear equations to produce a set nation changes is a linear model. 

of correction parameters; and 25 1^- Th^ method of claim 11 where the model for illumi- 

c) correcting at least one of the first and second images nation changes is a VigneUing model. 

using the correction parameters to remove distortion. 14. The method of claim 11 wherein geometric lens 

2. The method of claim 1 where the distortion includes distortion is modeled by the transform: 
geometric lens distortion. 

3. The method of claim 2 wherein geometric lens distor- 30 ^ > 
tion is modeled by the transform: ~ YT^^ixH^) ^' " T+T^ij^Ty^j 

I -t-Kjix^ -i- y^) ~ l+Kjix^ -i-y^) where (x,y) are the corrected pixel locations, (x^,yg) are 

distorted pixel locations, and is the distortion factor. 

where (x,y) are the corrected pixel locations, (x^y,) are A computer storage medium having instructions 

distorted pixel locations, and k, is a geometric len^ distor- stored therein for programming a computer to perform the 

tion factor. method of claim 11. 

4. The method of claim 1 wherein the correction param- 1^- method of claim 1 wherein step a) further 
eters correct a change in illumination along the at least one 40 comprises the step of performing motion estimation mod- 
motion trajectory. ^liiig using at least one of a translational motion model, an 

5. The method of claim 4 wherein illumination changes ^^ne motiori model, a bilinear motion model, and a per- 
are modeled with a linear model. spective motion model. 

6. The method of claim 4 wherein illumination changes 

are modeled with a Vignetting model. * ♦ ♦ * * 
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